Embeddings and Transformers¶
Goals of this lecture are:
- understand word embeddings
- fine-tune a BERT model for text classification
- visualize BERT embeddings
- understand multimodal embeddings from CLIP
- use CLIP for a image-to-text retrieval system
- implement a text-to-text retrieval system with the universal sentence encoder model
Structure of this Notebook¶
In this lecture, we begin by analyzing and understanding a pre-trained word2vec model that was used in earlier modern NLP systems before transformer models were released. We still use word2vec here to get a basic understanding of how words can be represented in neural networks and why transformer models have an advantage over word2vec embeddings.
After analyzing the word2vec embeddings, we fine-tune a BERT model on the provided German news dataset and visualize the BERT embeddings to get a better understanding of the differences.
The next major step after transformers for language modeling was the CLIP model, which combines text with images. We analyze the CLIP embeddings for different datasets and get an idea of why CLIP can be used for zero-shot classification of image data without fine-tuning the model on new data. We then use the CLIP model to find for given images, suitable news messages.
In the last part of this lecture, we implement a text-to-text retrieval system by first understanding why CLIP is not well suited for text-to-text retrieval, and then using the universal sentence encoder model for retrieving text passages.
This notebook will show you how easily these models can be used with the HuggingFace and SentenceTransformers libraries, which provide a high-level interface to the aforementioned models. This lecture is primarily about understanding the concept of embeddings rather than how to define your own architecture and train it.
First, however, we start by explaining the history and evolution of text representations for machine learning algorithms.
Reuqired libraries:
conda install numpy scipy
conda install scikit-learn
conda install numba
pip install umap-learn
pip install plotly
pip install 'transformers[torch]'
pip install sentence-transformers
pip install gensim
Overall Picture and Evolution Path of Word Embeddings¶
The conventional way of modelling documents in tasks like information-retrieval, document-clustering, document-classification, sentiment-analysis, topic-classification is to represent each document as a Bag-Of-Word-vector $$\mathbf{d}_i=(tf_{i,0},tf_{i,1},\ldots tf_{i,|V|}).$$ Each component of this vector corresponds to a single term $j$ of the underlying vocabulary $V$ and the values $tf_{i,j}$ counts the frequency of term $j$ in document $i$. Instead of the term-frequency $tf_{i,j}$ it is also possible to fill the BoW-vector with
- a binary indicator which indicates if the term $j$ appears in document $i$
- the tf-idf-values $$tfidf_{i,j}=tf_{i,j} \cdot log \frac{N}{df_j},$$ where $df_j$ is the frequency of documents, in which term $j$ appears, and $N$ is the total number of documents.
Independent of the values used, the BoW model represents sets of documents as 2-dimensional numeric arrays, which can easily be passed to conventional Machine Learning algorithms, such as Logistic Regression, SVM, MLPs etc. However, the major drawbacks of this approach are:
- the order by which terms appear in the document is totally ignored
- semantic relatedness of terms is not modelled
- BoW- vectors are very long and sparse
As sketched in the picture below, today all of these drawbacks can be circumvented by representing the words of the document by their word-vectors and passing the corresponding sequence of word vectors in order to either a CNN or a Recurrent Neural Network such as LSTM or GRU.
The figure above shows an overview of the training objective. During training, CLIP is incentivized to match each image in the current batch to the correct caption. Then the loss for both encoders is calculated and backpropagated. This pushes the representation ob both encoders onto a common latent space. All other examples in the current batch are negative examples, as the caption doesn't fit to the image. This is called contrastive learning. The goal of contrastive learning is to learn representations in which the embeddings of similar instances are close to each other, while different instances are further away.
(both pictures from https://openai.com/research/clip)
(img src: https://ai.googleblog.com/2019/07/multilingual-universal-sentence-encoder.html)